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ABSTRACT 

The report is a review of research results which 
concern sequencing (procedures for validating the hypothesized 
ordering of the subordinate tasks in a learning hierarchy) and 
practical applications of learning hierarchies in instructional 
design. Indirect versus /direct validation procedures are examined 
detail. (Author) 
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DEVELOPMENT OF 0PTIMAL~INSTRUCT10NAL SEQUENCES^ 

E. Ray Phillips 
University of South Florida 
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An optimal instructional sequence can be defined as one which maxi- 
mally facilitates achievement, transfer, and retention; requires the 
least amount of time for the learner to complete; and induces minimal 
anxiety and frustration .on the learner's part (Phillips, 1971). Two 

" questions arise logically from the formulation of the above definition. 
(1) Do optimal instructional sequences exist? (2) How are they 
determined and verified? The purpose of this paper is to revl^ the 
empirical evidence supporting the feasibility of the development of 
optimal instructidfi sequences and to explore the problem of developing 
and validating such sequences. / 

Learni ng Hierarchie s 



A learning" hierarchy may be characterized as "representing the most 

probab 1 e expec ta tlon of gr ea tes t positive tiransf er-f orr an ent ire sample 

of learners whom we know nothing more than what specifically relevant 
skills they start with" (Gagne, 1968). Thus,, it appears that, learning 
hierarchy theory can serve as a foundation for determining optimal instruc- 
tional iequencos. 



P^J^^^sontcd c.t the Symposium, "Measurement Aspects of Instruct- 
ional . Sequencing" , American Educational Research Association Annual 
Meeting, Chicago, April, 197A. 
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Numerous studies support the hlerarchlal structure of learning 
theory. DlVesta and Walls (1967) demonstrated positive transfer -from 
relevant "pre-utilization" training to the Maier two-string problem. 
Davis (1967) showed the effectiveness of transfer of previously learned 
verbal rules to swit<th-light problems, and a similar theme is developed 

, t • ■ 

\ 

by Overing and Travers (1966, 1967) in. studies pertaining to the probled 
of hitting an underwater target. Scandura and Wells (1967) demonstrated 
positive transfer effects from prior learning in concrete situations 
involving relevant rules in problems concerning mathematical groups and 
combinatorial topology. BaLtig (^968) cited evidence that the learning 
of paired associates is typically facilitated by prior discrimination 
learning on stimulus-terms and response-terms, as well as by prior learn- 
ing of stimulus coding responses. For a more extensive review of studies 
related to transfer of training, Schultz (i960) may be consulted. 

Another type of study from which evidence about learning hierarchies 
may be derived is one which attempts to try_out a total hierarchy in 
which the various levels of intellectual skills are to be learned in a • 
singre instructional s^^^^^ hypothesized that "an 

Individual will not be able .to learn a particular topic if he has failed 
to achieve any of the subordinate topics that "support it." This 
hypothesis was tested in several studies of the aforementioned typo. 
Gagne and Paradis. ■ (1961) studied transfer relationships within a 
learning hierarchy for the task of solving linear equations. They 
concluded that when particular subordinate skills required for new 
learning were present in the learner high positive transfer. resulted. 
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When they were absent, very low transfer took place. In a similar study, 
Gagne. Mayor, Garatens and Paradise (1962) measured the effects of • 
learning program variations upon achievement. The, results indicated that 
acquisition of subtasks ai: successively higher levels of the hierarchy 
was dependent upon pMor mastery of subordinate subtasks. The authors 
also found thlt when the mediating effects were examined for subtasks 
Intervening between higher and lower ones, proportions of achievement 
of higher subtasks indicated signif^.cant amounts of positive transfer. 
Several other studies (Gayne, 1962; Gagne, 1963; Gagne and Staff, 
University of Maryland Mathematics Project, 1965) yielded the same 
results reinforcing those of the previously discussed studies. In , 
general, the results of these studies indicate that new skills and 
knowledge emerge from lower order knowledge, and that there is a signi- 
ficant amount of positive transfer from each successive subordinate 
level to the next higher level in a hierarchical ordering of such levels. 

Sequ encing Instructional Materials 

Instructional dec;ign requires decisions about Structuring content 
and designing and ordering a set of instructional tasks. Gagne (1967) and 
Briggs (1968) have proposed the use of instructional sequences that require 
the learner to follow a specific route through a content structure, suggest- 
ing that optimal instructional sequences can be developed by sequencing 
materials according to learning hierarchies. 

Substantial evidence suggest that optimal learning sequences do, in 
fact, exist. Recent studies of sequencing (Brown, 1970; Niedepeyor, Brown, 
& Sulzcn, 1969) indicated that Ss using materials sequenced according 
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to, learning hierarchies performed reliably better than Ss using materials 
whose sequence was scrambled, relative to time to complete the instructional 
program, to errors made on the program, and to performance on a criterion 

* test of complex problem-solving skills. Brown (1970) concluded that when 

a sequence involves tasks that are complex, ordering of problem-solving 

. behaviors is an important factor in learning, even for bright and relatively 
mature learners. In suimnarizirig- research on sequencing mathematical tasks. 
Miller (1969) concluded that mastery of individual subtasks in a hierarchy 
can be achieved in several ways, including learning from randomly ordered 
sequences, b^^it that logical sequencing appeared best in terms of overall 

' efficiency and effectiveness. Roe (1962, p. A09) stated that "careful 
sequencing of "items has a significant effect on student performance, at 
least for programs of" some length and complexity.'* King (1970) described 
the above as key studies utilizing programs based on hierarchies and well 
controlled learning situations avoiding methodological weaknesses. 

Phillips arid Kane, (1973) conducted a study to investigate the effects 
of seven different sequences upon the overall efficiency of learning 
from programed mathematical majtcrials. No sequence maximally facilitated 
achievement, retention, and trl[insfer, and required less time to complete* 
However, based on the group means, students using materials sequenced 
via a hierarchy developed by the Walbesser - AAAS (1968) technique 
performed more efficiently than student using any of the other six 
sequences. 

Several studies. (Levin & Baker, 1963; Miller, 1965; Payne, Krathwohl, 
& Gordon, 1967; Roc, Case, & Roc, 1962) however, suggest that varying 



ERIC 



sequences of instructional stimuli that have high interdependency does not 
make much difference in effectiveness of instruction. However, some of 
these studies are plagued with design problems (King, 1970). Before seq- 
uencing instructional materials for use in classrooms, the effects 
of sequence upon time to achieve the terminal behavior, achievement, 
transfer, and retention should be investigated. ^ 



Hierarchy Validation Techniques 



Validatirtg a learning hierarchy jCF^fKit a simple undertaking. Many 
researchers (Ausubel, 1963; Bruner, 196A; Gagne, 1965; Glaser, 196A; and 
Suppes, 1966) have long recognized that sequence is a critical variable in 
learning. The learner begins with simple tasks and progresses to increas- 
ingly complex tasks. However, both Gagne (1968) and Pyatte (1969) have 
pointed out that determination of this hierarchical ordering of suljtasks 
from simplest to most complex is a major problem. 

Gagne and Paradise (1961) W-ere pioneers in learning hierarchy valid- 
ation. Their approach was direct validation based ort learners' responses 
to a programmed learning-sequence and criterion tests administered iramcd- 
iately after the instructional program to establish pass-fail patterns for 
each component of the learning hierarchy. Consider the simple two-level 
hierarchy in Figure 1. » 



II 



Higher Level Task 
(more complex) 



Lover Level Task 
(less complex) 



Fig;ure 1. A Simple Two-level Hierarchy 
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Gagne's validation procedure was based on the assumption that task I 
must be mastered before task II can be mastered-. Failure on task I would 
automatically produce failure oh task II. Using + and - to represent pass 
and faij respectively , there are foiir jiossible pass-fall relationships 
which can be observed: (++) , (+-) , (— ) , (-+) • For example, the first 
relationship signifies that the learner passed (performed to criterion) both 
task I and task III Only l^he relationship (+-) l3 in direct contradiction 
of the theory and indicates a flaw in ordering. The relationship (-+) 
(passed lower level task but failed higher level task) indicated a weak- 
ne'ss in the instructional program but provides no information concerning 
the validity of the hierarchy. 

^ To validate a hierarchy Gagnc analyzed the pattern of responses of 
each transfer in the hierarchy. That is, he constructed a contingency table 
of the observed responses to a higher level task and the task immediately 
prerequisite to it as illustrated in Figure 1. He calculated the following 
ratio to determine the degree of validity of the hierarchy. 

PfoportioH of Positive Transfer (P+) (-t-f) + (— ) 

i (++) + (— ) + (+-) 

Perfect validity would be indicated by a ratio of 1.00. If all learners 
contradicted the theory, having observed patterns (+-) , then the ratio 
would be zero. Thus, P+ is bounded above and below by 1 and 0 respectively. 
The degree of validity of any hierarchy is measured by P+ with the lower 
limit of acceptability for P+ being .90. 

Phillips and Kane (197''t) investigated the efficacy of this ratio when 

■ I . r 

applied to test data alono. Usinf; Gagne's task analysis a learning bier- 



archy for the computational skills of whole number addition was constructed. 
Based on the hypothesized ordering of the subordinate levels, a test was 
constructed to assess mastery at each level. A second test utilizing a 
random ordering of the same items was constructed. Both tests were adrain-^ 
istered to a large sample of elementary school children in grades 3 througtiT 
6 ifi otlier to obtain a wide range of ability levels. The proportion of 
positive transfer between ad.iacent items on both tests was computed using 
Gvigne's formula. The proportions between adjacent items on both tests were 
above .90, except in two" instances. Thus both, the hypothesized and thq 

i 

random hierarchies were validated by this procedure, The authors conclluded 
that prior educational experiences confounded the issue of positive transfer 
when considering test delta alone. ■ 

Walbesser (1969) has refined and extended Gagne's approach to hier- 
archy validation.- Task analysis is used to gerierato hierarchies of sub- 
ordintae subtasks. . Learning sequences are designed to correspond to the 
hypotheFized" hierarchies. Pass-fail contingencies are used- to test 

- - • ^ s" ~ 

dependency of each individual task on its immediate prerequisite subtasks. 
The Staff for AAAS pointed out that a high proportion of positive transfer- 
Gagne's statistics - is a necessary but not a suffici^t condition for a 
valid hierarchy. Using the pass-fail relationijhips defined by Gagne , the 
AA/\S defined the following three ratios: 

(1) Consistency ratio = 

(++) + (+-) 

(2) Adequacy ratio =■ (++) . ' ' 

(^-f) + (-+) 

(3) Completeness ratio (-H-) 
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tio (1) is a measure of how consistent the data are with the hypothesized 
dependency. Ratio (2) is a measure of the adequ^.'-.y of the Identified sub- 
ordindite tasks. Ratio (3) is a measure of the effectiveness of instructional 
m^erlals designed to bring about learning. In the development of Science — A - 
Process . Approach , The AAA5 (1968) has considered high consistency, adequacy, 
and completeness ratios as the necessary ati^ sufficient Set of characteristics 

■ ■ .1 

for a valid learning hierarchy. No significance test: .has been {developed for 

either Gagne's ratio or those defined by the AAAS. Eisenbergand Walbcsser (1971) 

have further refined this validation technique which includes tljie use of six 
different ratios. • 

Cox, and Graham (1966) used the Guttman Scalograw Analysis (19A4) to dev- 
elop a sequenclally scaled achievement test. Essentially, the Guttman tech- 
nique (Torgerson, 1958) orders items such that from knowledge of a learner's 
total score, his response pattern to the set of items can be predicted. The 
coefficient of reproducibility which Is given by one minus the ratio of the 
total number of errors to total number of responses 

Rep = 1- total number of. errors 

total number of responses 

<f 

indicates the degree to which a set of items forms a perfect scale. Error is 

■ . / - ., 

defined as instances where a subject passes a higher level item after failing 
a lower level prerequisite item., Guttman suggested .90 as an acceptable lower 
limit for Rep. Cox and Graham reported a reproducibility coefficient of .97 
for their final arrangement of items and thus concluded their hierarchical 
ordering as valid. 
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Several investigators have attempted to use item difficulty as a means 
of validat,ing a hierarchy. Studies by Stoker and Kropp (196A) and Herron 
(1965) showed that the cognitive skills in the Taxonomy (Bloom, 1956) did 
form a hierarchy. Items assessing skills Mgher up in the Taxonomy were 
more difficult than those at lower levels in the Taxonomy . Stoker, Kropp and 
Bashaw conclijded that their results based on item difficulty validated the 
Taxonomy . Miller and Phillips (197A) used item difficulty irt cc^junction 

with the Walttesser technique to validate a hiererchy for the computational 

J . 
skills of' rational numbe^ subtraction. 



There are several methods of hierarchical analysis reported iti the liter- 
ature which are used in the generation of hierarchies rather than the valid- 
ation of d.eductively analyzed hierarchies. McQiiitty (1960) developed a proced- 
ure ifor determining if a hierarchical structure underlies a set of items. He 
began with a large item pool with no a priori assumptions regarding the relat- 
ionship between^ il:«ms^ in tefihs-bf comp^ The procedure consisted of 
coinbining pairs of items or variables ^hich correlated highest with each other 
to form new items. This procedure is repeated until one pair of items remains. 
When items arc plotted on a linear scale and successive pairs are :;onnected 
by lines the resulting diagram has a hierarchical structure. Smith (1968) 
employed McQuitty's method to investigate the hierarchical model underlying 
Bloom's Taxonor.iy . In general, Smith concluded that his analysis supported the 
Taxonom y rationale of a cumulative and hierarchical continuum of cognitive 
procesKos. McQuitty (1966) has further refined and improved his method of hier- 
archical syndrome analysis. Multiple Sralogram Analysis, (MSA) developed by 
. Linpoes (1963) can be used to generate a hierarchical ordei i.ic from a largo 
set of items for which there are no a i )riori assumpllon.s made resardinc order. 
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The procedure essentially acco^lished the same, goal as the Guttman Tech- 
nique with built In- controls againsT spuriously high' reproducibilities as , 

^J, • ■ . ■■ 

a function of extreme marginal values. This procedure tends to produce 

several bi^anches which have very llttl'e In common with one another. Resntck 
and Wang ll969) have used tfSA to generate various hierarchies. The methods 
..developed by McQultty (I960-) and Lingoes (1963) , as well as several other 
methods of hierarchical analyses outlij^ed by Torgerson (1958) , are not 
readily lidaptablc for use in validating^ hypothesized ^hierarchies . In thesa 
methods the data rau^ speak for themselves with no a priori assumptlohs 
concernlng^order. 'Carroll (Resnick Ld Wang, 1969) has developed a hierarchy 
validation \proccdure based on the correlation between items or subtasks. , This 
method, Uk^ those of Gagne and Walbesser, is based on pass-fall, contingency 
tables for. all possible pairs of items in the hierarchy arid the phi-phlmax 
coefficients with a built-in control a^i^st artificial inflation due to 
' extreme pass-fall .rat^.~"ln discussing different approacttfes to hierarchy , 
validation, Rcsnick and Wang (1969) citeu no^^tudies utilizing CajrroU's 
method. Grib and Rimoldl (1960) developed a procedure for comparing two 
patterns of using the index of agreement. Listing each subject's responses • 
to a set of items produces an observed matrix of responses With rows corres- 
ponding to items. An expected matrix can be formed based on^an operational 
definition of what response patterns are expected from a given set of items. 
The only restriction on the expected nfatrlx is that the subjects t otal score 
on the Observed pattern. These two patterns' can be compared and an index 
showing the amount of agreement between the tw(J pattern^* can be computed. 

■ Phillips and Kane (1973) conducted a stud/ to investigate the efficacy of 
7 procedures for ord'cri^g the' subtasks in a learning sequence. An initial 

\ 
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hierarchy for rational number addit^ion was constructed. A test to assess 

* ' , • ■ ' ' • 

i mastery at .each of the 11 levels .v,i3>: admi:nistered to 163 elementary school. 



children and the pass-fail relatioflships*^ix?ere analyzed using the following 

.- . '■ u ' ■ 

'•. procedures: 07 Item difficulty (Nunnally, 1967), (2) The AAAS approach 

;(AAAS Commission on Scieijce Education, 1968), (3) The Guttman Technique 

- /(Torgerson;. 1^58) , (4) Pattern analysis (^imbidi and cirive, 1960) , and 

' "(5)' Correlation analysis (PhilTips, 1971). For a detailed description of 

, ■ . •■ / ' 

these' procedures, see Phillips (1971). The subtasks were also sequenced 

< , * 

according to the "usual" textbook sequenpe and randomly. To exami-ne the 

• t 

adequacy of each validation procedure, pro^ranjraed instructional materials 

were sequenced according to the hierarchy igendra.Ved |b^ (lach procedure.., 

■ • \, ; ■ • ■ ■ . . • ■• ■ . ■ 

» Subjects were randomly assigned to the 7 tlfeiitir.ents cvnd groups were compared *^ 

oVi achievement, transfer, fete^tici^n, and .4;ime to com^jLete the instructional 

sequence. No sequencev^.ma>:;imally fitcilitated achievcn^ent , retention, and 

transferV^and required less^ time to cXmplete. However, based on group means, 

the AAAS procedure yielded the best sequence cwe.rall , and^ was judged the best 

hierarchy validation technique of the seven irndfef investigations. 



/ 
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Conclusions 

• ■ 

In reviewing the literature, one finds substantial evidence to support 
the general theory of the hierarchical ^tructure of knowledge (studies cited 
previously). There seems to' be. little doubt that new skills and knowledge 
emerge from lower order knowledge, and that there' is a significant amount of 
of positive transfer from each successive subordinate level to the next hlgh.-r 

level in a hlerarcliical ordering of such levels.^ T^^^^ on. .s,eauen_cins.„ , „. . 

instructional materials reviewed suggest that the sequence of subordinate 
tac.ks in a learning hierarchy, after sufficient validation, should describe 
a teaching program that will effectively accomplish the instructional object-, 
ives. Th'at is, an instiuctional sequence based on the levels in the hierarchy 
will represent an optimal route for a«:qulsition of the terminal task by a 
sample of learners. Kence, it appears that the development of optimal 
instructional sequences becomes essentially a problem of validating a learn- 

.1 

ing hierarchy. 

The question of validating the ordering of the levels in a hierarchy is 
a compli-x and illusive probl|em. Gagne (1968)' stated that various methods have 
been tried but. none seem entirely satisfactory as yet? Several of these tech- 
niques were reviewed in this paper; and undoubtedly, others are beirir developed 
and tested. It seeras, impcrattve that efforts to develop more efficient pro- 
CfdurcH for validating it^arnlng hierarchies be continued. 
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Sovcral questions have been raised regarding Gagne's method of empirical 
validation ofj the hypothesized ordering of a set of subordinate tasks (Wal- 
bcs.^cr, 19G8; Phillips f. Kane, 1974; Phillips, 1971; Wilte, 1971). These 
Include. sraaJl saniplr r,i/.i-, imprecisely defined use of tar,k analysis, no 
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/ significance test for the proportion of positive transfer ratio, and errors 

of measurement. Vmite (1973) has suggested two lines of further learning * 
hierarchy research.. (1) Use Gagne's. model and try to minimize the above 
problems. (2) Use White's (1973) more rigorous model. White's method 
essentially states, for a set of k subtasks, teach them In all kl possible 
ordering^; to different groups. Based on the performance of the learners In \ 

each _group, one determine^. -the -'bast" ofderiug* This ordering represents 

a valid hierarchy. Obvious difficulties with this method, of course, are the 
large number of experimental groups that would be Involved, the number of 
subjects needed, (and the time involved in this type of, validation. When 
the number of sub\^sks is large. White suggested that the research ^must re- 
sort back to Gagne'Vorlglnal design with some recommended changes. One of 
the more important of elwi^commendatlbns is to replace Indices such as 
Gagne's "proportion of positive transfer", the ratios proposed by Walbesser 
and Guttman's index of reproducibility by a statistical test of hierarchical 
dependence. White and Clark (1973) report the development of such a test. 

Reconuncndat io ns 

Validating a learning hierarchy by either of White's suggested procedures 
is a tedious and costly undertaking. In either case, one must teach a sequence 
of tasks to a large sample of learners; and with the rigorous model, kl 
sequences must be presented to different groups of learners. Based upon the 
performance ol the learners in the experimental groups, the. best ordering 
is determined as the valid sequence. Thus, it seems that l^^hite would define 
a valid learning hierare.ljy similar to Kane, McDanlcl and Phillips (1971). 
That is, a valid hierarchy if-, one wliich yields an optimal Icirning sequence. 

ERIC 
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Phillips (1971) labeled validation >t&€ffiniques that are based on test 
data'alone are called indirect. It seems that the development of. less 
expensive indirect validation techniques could greatly improve the use of 
learning hierarchies in designing instructional sequences. 

A hypothetical case Is considered below to illustrate how one might 
use an indirect procedure to validate a l&arning hierarchy. Supple 20 
subtasks are to be ordered hierarchically, and for simplicity's sake item \ 
difficulty (Nunnally, 1967) will be used as the basis of ordering (the author 
is not £issuming nQr._proposlng that item -difficulty is an adequate validation 
procedure) . 

1. Construct a test with 3 items for each of the 20 subtasks. (Pass 
for each subtask will be correct responses to 2 of the 3 items four 
that subtask.) " 

2. Administer the test to a large sampl^ of subjects to include a wide 
range of abiHty and achievement levels. 

3. From the test result^;, calculate the item difficulty for each of 
the 20 items and order from highest to lowest. This will represent 
the "valid" ordering of the 20 subtasks. '''^ 

Regardl-ess of the validation technique used with the three steps above, 

one must have some evidence of the adequacy of the technique. That is, did the 

technique employed yield a valid hierarchy? Such evidence may be obtained in 
the following manner: 

1. Design instructional materials for each of the 20 subtasks. 

2. Sequence these materials according to the hierarchy generated by 
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the indirect procedure under scrutiny (in this case item .diff- 
iculty, i ' • 

3. Teach a large sample of learners selected at random using the 
above sequence of ins^uctional materials. 

4. Sequence the instructional material for each of the 20 subtasks 

in several other orderings generated by direct or indirect methods as 
well as sequence ordered from logical analysis of the content and 

« 

sequences deliberate^ ordered in a manner which one believes would 
interfere with learning. , . | 

I 

i 

5. Randomly assign a large number of students to these groups. 

■ / ^ - 

6. At the completion of the instructional program compare each group 

on achievement, retention, transfer , /attitude toward the progra^J, and 

V . . . ( 

time needed to complete the program. 



7. If the indirect procedure produces/ an optimal learning sequence, one 
concludes the procedure is adequate for generating a valid hierarchy 
(Keep in mind more than one valid hierarchy could be generated from 
the same 20 subtasks) . / 

* ■ • .1 

.( 

. • . I . 

Now that the procedure has been put to the ultimate test of acti^ally 
sequencing instructional materials based on the hierarchy generated,, one 

« 

can apply the indirect procedure to a different set of subtasks with some 
confidence. That is, there's no need to rediscover the wheel each time a 
hierarchy is to be validated. The indirect procedure can be applied to test 
data alone without going through a costly instructional program to validate 
directly. 



Hopefully new hierarchy validation technique will be developed which may 
be adaptable to the indirect approach to validating a learning hierarchy. 
With the development' of adequate hierarchy validation techniques, the poss- 
ibility of the development of optimal instructional sequences becomes a very 
real one. 



\ 
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